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ABSTRACT 


A  method  of  linking  compatible  straight  edges  is  pre¬ 
sented  and  discussed.  The  linking  is  based  on  information 
about  the  geometrical  configuration  of  the  edges,  the  simi¬ 
larity  of  the  gray  levels  on  their  object  sides,  and  the 
similarity  of  their  object  sides  with  the  line  joining  their 
endpoints.  Three  figures  of  merit  are  defined  for  evaluating 
pairs  of  segments  for  possible  linking.  Examples  are  shown 
of  applying  the  method  to  high  resolution  aerial  photographs. 
Results  indicate  that  cultural  features  such  as  roads  and 
buildings  can  be  extracted  and  that  a  significant  reduction 
in  the  complexity  of  the  image  description  can  be  obtained. 
This  approach  should  be  especially  useful  for  defining  the 
degrees  of  compatibility  of  pairs  of  edges  in  a  relaxation 
scheme  for  classifying  linear  feature  segments. 
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1.  Introduction 


Cultural  features  often  contrast  with  their  surrounds, 
and  are  usually  bounded  by  sharp,  locally  straight  edges.  As 
a  preliminary  step  towards  the  recognition  of  such  objects, 
the  straight  edges  can  be  extracted  [1,2]  and  clustered  into 
compatible  pairs  (i.e.  pairs  of  edges  having  compatible  object 
sides  that  can  be  merged  together) .  The  clustering  process 
is  the  subject  of  this  report. 

Line  linking  techniques  have  been  studied  since  the  early 
days  of  this  field.  The  literature  is  too  large  to  cite  here 
in  full.  Partial  surveys  may  be  found  in  [3-6].  In  general, 
clustering  must  take  into  account  information  from  the  picture 
in  the  regions  around  the  edges.  For  example,  a  roof  of  a  house 
usually  has  a  uniform  gray  level  and  thus  it  is  reasonable  to 
expect  that  the  object  sides  of  the  compatible  pairs  have 
similar  gray  levels.  The  process  to  be  described  here  deals 
with  lines  extracted  from  an  image  together  with  information 
about  the  object  side  of  each  line  (e.g.,  bright  objects  or 
dark  objects  with  respect  to  the  background  scene) . 

The  process  accepts  as  input  a  set  of  ordered  pairs  of  line 
endpoints,  and  the  original  gray  level  picture.  It  attempts  to 
find  pairs  of  lines  that  are  geometrically  compatible  and  that 
obey  relations  involving  similarity  of  object  side  gray  levels. 
The  approach  that  we  are  taking  here  toward  a  global  represen¬ 
tation  of  the  line  segments  in  a  picture  begins  with  finding 
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potential  linking  segments  between  the  ends  of  line  segments 
in  the  picture,  and  measuring  certain  properties  of  the  seg¬ 
ments  and  the  linking  segment.  Once  we  have  found  strong 
linking  segments,  our  approach  proceeds  toward  a  global  repre¬ 
sentation  of  the  line  segments  by  linking  them  together  or 
merging  them.  The  merging  process  begins  by  choosing  a  few 
linese  closest  to  each  end  of  a  given  line  as  candidates  for 
merging.  Then  a  figure  of  merit  based  on  the  similarity  of  the 
object  side  gray  levels  and  angle  between  the  lines  is  calculated 
for  each  pair.  Three  figures  of  merit  are  suggested,  based  on 
successively  more  knowledge  about  the  scene. 

This  report  suggests  three  figures  of  merit  defining  the 
mutual  support  of  two  linear  features  for  merging.  In  what 
follows  the  compatibility  conditions  will  be  derived  and  then 
the  figures  of  merit  will  be  explained. 
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.  Compatibility  conditions 

Each  pair  of  lines  should  satisfy  certain  conditions  in 
order  to  be  accepted  as  a  candidate  compatible  pair.  In 
general  these  conditions  are: 

a)  Similarity  of  gray  level  of  a  strip  connecting  their 
ends  with  respect  to  the  gray  level  on  the  object  side  of  the 
pair. 

b)  Similarity  of  gray  level  of  strips  on  the  object  sides 
of  the  pair. 

c)  Conditions  on  the  geometrical  configuration  of  the  pair 
of  lines. 

d)  Conditions  on  the  angle  between  the  lines. 

e)  Object  gray  level  measures. 

In  what  follows  each  of  the  above  measures  will  be  discussed 


in  more  detail. 


2.1  Similarity  measures 

Referring  to  Figure  1  the  measure  of  similarity  between 
line  A  and  lines  A^,A2,A3,  etc.  is  defined  as 

Sil=l  g'gi.! 

where  g  and  g^  are  the  average  gray  levels  along  strips  on  the 
object  sides  of  lines  A  and  A^,  respectively.  Similarly  the 
measure  of  similarity  between  a  linking  line  and  the  pair 
of  lines  A,A^  is  defined  as 
Si2= I 9ci“9 I 

and 

si3=lgci-gii 

If  Si;L,  Si„»  and  Si3  are  small  for  a  pair  of  lines,  then  that 
pair  is  compatible  on  the  basis  of  similarity  of  the  gray  levels. 
To  define  a  figure  of  merit,  a  linear  function  as  shown  in 
Figure  2  is  defined  for  each  measure  of  similarity.  According 
to  this  linear  function  when  the  above  differences  are  zero  the 


figure  of  merit  d^k  (k=l,2,3)  is  equal  to  1  and  when  the  differ¬ 
ence  is  greater  than  the  maximum  allowable  level  of  similarity 


dik  is  equal  to  zero.  In  other  words 


dil  "  { 


0  if  S...>S 

il  max 

1-S. ,/S  if  S.  ,>Sm= 

il  max  il  max 


where  Smax  is  the  maximum  allowable  similarity  level. 


Similarly  for  d^2  and  d^3  we  have 


{0  if  S.,  >S 

lk  max 

1-S../S  if  S..  *S 

lk  max  ik  max 

1  if  the  distance  between  the  end  points 

is  less  than  the  width  of  the  strip  used 
for  calculating  the  average  gray  level. 

for  k=2  and  3.  In  cases  where  the  distance  between  the  end 

points  is  less  than  the  width  of  the  strip  used  in  calculating 

the  average  gray  level,  the  calculation  is  not  reliable  (the 

number  of  points  used  in  this  calculation  is  too  limited) .  In 

these  cases  since  the  distance  is  small  the  benefit  of  doubt  is 

given  and  the  value  of  the  function  is  set  equal  to  1. 

The  total  figure  of  merit  due  to  similarity  is  defined  as 


D .  =d .  .  +d .  _+d . 
i  il  i2  i3 

The  maximum  possible  value  for  Di  is  3,  and  the  lowest  value  is  0. 


2.2  Geometrical  measures 

Geometrical  conditions  are  important  in  making  two  lines 
compatible.  Figures  3a  and  3b  show  examples  of  geometrically 
compatible  and  incompatible  pairs,  respectively.  The  plus 
sign  refers  to  the  object  side  of  the  segment.  It  is  clear 
that  objects  cannot  be  built  with  the  lines  shown  in  Figure  3b. 

To  differentiate  between  geometrically  compatible  and  in¬ 
compatible  pairs  certain  constraints  on  the  geometrical  loca¬ 
tions  of  the  end  points  are  necessary.  Here  we  allow  a  large 
gap  between  the  pairs  if  the  lines  of  the  pair  are  long,  and 
vice  versa.  Having  this  in  mind  we  will  assume  that  the  gap 
should  be  less  than  the  other  distances  connecting  the  other 
ends  of  tne  pair  of  lines.  Figure  4  shows  examples  of  different 
orientations  of  pairs  of  lines.  Referring  to  the  object  side 
and  end  conventions,  the  ratios  of  the  distances  are  summarized 
as  follows: 

For  case  (a) 

R2=a2bl^albl  R3=a2bl^alb2 

For  case  (b) 

Rl=albl/alb2  R2=albl/a2b2  R3=albl/a2bl 

For  case  (c) 

Rl=alb2/a2bl  R2=alb2/a2b2  R3=alb2/albl 

For  case  (d) 


R2=a2b2/a2bl 


R3~a2b2//alb2 


In  order  to  calculate  these  ratios  the  ends  and  the  sides 


of  the  lines  should  be  consistently  labeled  (see  (7]). 

Assume  that  for  each  neighboring  line  (i)  these  ratios 
are  called  R^.  Then  the  geometric  measure  for  each  ratio 
is  defined  as: 


fl  if  Rki<;L 
lo  if  Rkisl 


where  k=l,2,3  and  i  is  the  label  of  the  neighboring  line. 

The  overall  geometrical  measure  is  defined  to  be 

G  .  =G . . • G . ~ • G . Q 
i  il  i2  i3 

According  to  this  definition,  if  the  geometrical  requirements 
are  satisfied  G^=»l,  otherwise  Gi=0 . 

By  examining  these  ratios  it  will  be  clear  that  the  smaller 
the  ratio  tne  better  will  be  the  geometrical  compatibility  of 
the  lines.  Using  this  fact,  another  more  restrictive  definition 
for  the  geometrical  measure  can  be  obtained. 

To  define  this  geometrical  measure,  let 


r  3 


2  Rki/3 

if 

Rki<;L 

«  k=l 

1 

if 

Rkisl 

for  k=l,2,3 

Figure  5  shows  the  suggested  linear  function.  According  to 
this  figure,  as  the  sum  of  these  ratios  decreases  with  respect 
to  the  minimum  value  m=min(r^),  the  geometrical  measure  decreases 
too.  The  geometrical  measure  H.  is  defined  by 


l-(r.  -m)/(J  -m)  il  vl 
0  if  ri=l 


2 . 3  Angle  function 

The  angle  between  geometrically  compatible  candidates  is 
also  an  important  factor  in  finding  compatible  pairs.  First, 
the  angle  between  the  geometrically  compatible  candidates  is 
calculated.  Figure  6  shows  examples  of  the  calculation  of 
angles  between  two  geometrically  compatible  lines.  In  this 
figure  0A  and  @B  are  the  angles  of  lines  A  and  B  with  respect 
to  the  horizontal  axis.  In  this  convention  the  angle  between 
two  collinear  gometrically  compatible  lines  is  180°.  Referring 
to  Figure  7,  the  angle  between  line  A  and  compatible  candidate 
A^  is  designated  by  0^.  As  shown  in  this  figure,  the  smaller 
the  angle  between  compatible  candidates ,  the  stronger  the  degree 
of  compatibility  between  the  lines.  To  express  this  situation, 
a  linear  function  is  again  proposed.  Figure  8  shows  the  pro¬ 
posed  angle  function.  This  linear  function  is  normalized  with 
respect  to  the  smallest  angle  0  between  the  lines  in  question. 
We  assume  that  the  angle  between  two  compatible  candidates  can¬ 
not  be  less  than  0^  {usually  25°),  and  cannot  be  greater  than 
27r-0£.  If  0g  is  less  than  0^  the  next  greater  angle  will  be 
used  for  8  .  If  no  angle  with  the  above  condition  is  found  the 
angle  function  is  set  equal  to  zero  for  all  the  compatible  can¬ 
didate  lines.  In  summary,  the  linear  angle  function  is  defined 


as  follows: 


i 


2. 4  Object  gray  level  function 

The  given  object  gray  level  or  estimated  object  gray  level 
can  also  be  used  in  defining  the  figure  of  merit.  This  measure 
can  be  incorporated  into  the  figure  of  merit  in  such  a  way  as  to 
yield  a  high  value  for  the  compatibility  of  object  lines  and  a 
low  value  for  non-object  lines.  The  gray  level  average  of  the 
object  can  be  automatically  estimated  as  follows: 

1)  Calculate  the  average  gray  level  in  a  strip  on  each 
side  of  each  line  segment. 

2)  Sort  the  line  segments  in  decreasing  order  of  length. 

3)  Select  the  longest  p%  of  the  lines  (usually  5%) . 

4)  Calculate  the  average  gray  levels  of  the  brightest  (or 
darkest)  sides  of  the  lines  selected  in  step  (3) . 

The  average  gray  level  calculated  in  this  way  can  be  accepted 
as  a  good  estimate  for  the  typical  gray  level  of  the  objects. 

In  a  case  where  the  scene  is  not  illuminated  uniformly,  one  can 
estimate  the  typical  gray  level  average  of  p%  of  the  longest 
lines  if  they  are  distributed  geometrically  in  different  loca¬ 
tions  on  the  scene. 

If  the  differences  between  the  estimated  gray  levels  of 
the  two  lines  in  question  and  the  average  object  side  gray 
level  is  small,  the  two  lines  should  reinforce  each  other  to 
yield  a  high  figure  of  merit.  Otherwise,  they  should  punish 
each  other  in  order  to  yield  a  low  figure  of  merit.  A  linear 
function  is  defined  for  this  purpose,  as  shown  in  Figure  9. 


In  this  figure  g&  is  the  estimated  gray  level  average  and  g^ 
is  the  gray  level  average  of  a  small  neighborhood  along  the 
object  side  of  line  i.  smax  is  the  maximum  allowable  level  of 
similarity.  In  summary  the  gray  level  measure  for  each  line 
is  defined  as  follows: 

1  if  0  &  I  cr  -q  •  I  &S 

|ye  yi 1  max 

f  (g .  )  =  ^  2-  I  g  -g .  I  /S  if  S  <|q  -q  •  I <2S 

|ye  yi!/  max  max  |ye  yi 1  max 

.0  if  I q  -q .  I  >2S 

v  Iye  max 

Using  the  above  definition  for  each  line,  the  mutual  gray 
level  strength  between  line  i  and  line  j  is  defined  as 

F(gij>  -  £(gi>-£(gj> 


3.  Figures  of  merit 

Three  figures  of  merit  are  defined.  The  first  one  uses  the  simi¬ 
larity  and  geometric  measures  and  the  angle  function.  The  second 
uses  the  information  used  in  the  first  one  as  well  as  the  object 
mutual  gray  level  strength.  The  first  figure  of  merit  is  de¬ 
fined  as 

FM1  =  Max (MS) 

where  the  merging  strength  MS  is  defined  as 

MS=D  .  F(0.)G. 
r  11 

Here  i  is  the  label  of  the  neighboring  line. 

This  definition  simply  multiplies  the  similarity  measure 
and  the  angle  function  for  the  line  in  question  and  other  neigh¬ 
boring  lines  and  the  binary  function  of  the  geometrical  measure. 

The  maximum  value  of  the  figure  of  merit  is  3  when  a  neighboring 
line  has  the  same  object  side  gray  level  average  as  the  linking 
line  and  the  given  line,  and  also  makes  the  smallest  angle  with 
the  given  line. 

The  second  figure  of  merit  uses  information  about  the  given 
or  estimated  object  gray  level.  This  is  again  defined  as 
FM2  =  Max (MS' ) 

where  the  merging  strength  MS'  is  defined  as: 

MS'  =  Dj_  F(0  ±)  Gi  F  (g  j  i) 

where  j  is  the  label  of  the  given  line  and  i  is  the  label  of 
the  neighboring  line.  Here  again  the  maximum  value  of  the  figure 
of  merit  is  3,  when  the  conditions  stated  for  the  first  defini¬ 
tion  are  satisfied  along  with  the  condition  that  the  differences 


4 .  The  algorithm 

1)  Read  in  the  data  for  each  line  in  the  form  (x^ ,x2 ,y2) 

where  are  the  co-ordinates  of  the  first  end  point  and 

*X2'y2*  are  co-orc^nates  the  secon<i  end  point  of  the  line 

2)  For  end  "1"  of  each  line,  find  the  closest  N  neighboring 
end  points.  Record  the  labels  of  these  lines.  (N  is  taken  as  3 
in  the  program  to  keep  computational  time  reasonable . ) 

3)  Calculate  the  figure  of  merit  and  record  it  along  with 
the  corresponding  ends,  line  label,  and  label  of  the  compatible 
line.  Mark  this  line  and  the  compatible  line  so  that  they  will 
not  be  processed  again. 

4)  Choose  the  other  end  of  the  compatible  line  found  in  (3) 
and  continue  the  process.  If  no  compatible  line  is  found  go  to 
(5)  . 

5)  Choose  end  "2"  of  the  line  started  with  and  go  to  (2) . 

If  end  "2"  is  already  processed  go  to  (6). 

6)  Continue  the  above  process  for  the  other  line  segments. 

The  compatible  lines  are  chosen  in  a  serial  manner,  that  is 

once  a  compatible  line  is  found  the  process  jumps  to  the  other 
end  of  the  compatible  line  and  continues  the  process.  The  algo¬ 
rithm  can  also  be  adapted  to  operate  in  a  parallel  fashion.  The 
width  of  the  strip  used  in  calculation  of  the  average  gray  level 
is  4  pixels.  This  value  is  selected  for  reasons  involving  the 


resolution  of  the  scene 


Examples  and  discussion 


The  algorithm  presented  here  performs  well  on  several 
input  data  sets.  The  straight  edges  in  the  images  were  ex¬ 
tracted  using  an  iterative  enhancement  technique  [2]  and  the 
resulting  lines  were  used  as  input  to  the  pairing  program. 

Figure  10  shows  one  input  image  and  the  set  of  lines  extracted 
from  it  is  shown  in  Figure  11.  Figure  12  shows  the  histogram 
of  the  first  figure  of  merit.  The  maximum  level  of  similarity 
is  set  to  be  10  in  calculation  of  the  figure  of  merit;  this  is 
quite  a  tolerant  value.  Figures  13  to  16  show  the  result  of 
line  linking  with  different  thresholds  {2.3,  2.0,  1.3,  0.1) 
on  the  value  of  the  figure  of  merit.  In  these  figures,  the  lines 
that  have  at  least  one  compatible  pair  within  the  threshold  of 
the  figure  of  merit  are  shown.  It  is  clear  that  the  process  is 
rather  stable.  The  best  threshold  is  around  2  (probability  of 
merging  is  equal  to  0.7). 

The  effect  of  changing  the  maximum  level  of  similarity  has 
also  been  studied.  When  this  level  is  changed  to  12  or  15, 
the  results  are  shown  in  Figures  17  to  26.  The  effect  of  in¬ 
creasing  the  maximum  level  of  similarity  is  to  modify  the  histo¬ 
gram  of  the  figures  of  merit.  Comparing  the  results,  it  is 
apparent  that  no  major  change  results  when  the  maximum  level  of 
similarity  is  changed.  Figure  27  shows  another  image,  and  Figures 
28  to  33  show  the  results  of  using  the  process  on  this  scene. 


Another  series  of  experiments  has  been  performed  using 
the  second  figure  of  merit.  Figure  34  shows  the  histogram  of 
the  second  figure  of  merit  for  the  first  scene.  The  results  of 
line  linking  using  different  thresholds  are  shown  in  Figures  35 
to  38.  It  is  seen  that  the  second  method  of  defining  the  figure 
of  merit  also  has  the  property  of  cleaning  the  non-object  lines 
(noisy  lines) .  Figure  39  to  43  show  the  results  for  the  second 
scene. 

Examination  of  the  output  with  different  thresholds  on  the 
figure  of  merit  shows  that  one  can  choose  the  lowest  nonzero 
threshold  without  causing  trouble. 

A  similar  series  of  experiments  has  been  performed  using  the 
third  figure  of  merit.  No  major  change  has  been  observed  and 
in  most  cases  exactly  similar  results  have  been  obtained. 

The  effect  of  increasing  ^nax  in  the  second  figure  of  merit 
has  also  been  studied.  As  we  increase  this  level,  the  probabi¬ 
lity  that  non-object  lines  are  linked  is  increased.  The  results 

of  increasing  S  to  12  and  15  are  shown  in  Figures  44  to  54. 
max 

The  linking  algorithm  in  this  paper  can  be  criticized  for 
taking  too  local  a  view  in  determining  the  best  matches  from 
the  three  closest  neighbors  only,  although  the  program  can  con¬ 
sider  more  than  three.  It  might  happen  that  a  pair  of  lines 


that  are  mutually  best  partners  should  not,  in  fact,  be  linked. 
This  problem  can  be  overcome  by  increasing  the  number  of  closest 
neighbors  in  calculating  the  figure  of  merit.  The  results 
show  that  reliable  merging  can  be  obtained  by  considering  three 
neighbors  only.  The  results  also  suggest  that  one  might  use 
such  algorithms  to  define  the  degrees  of  compatibility  of  the 
line  segments  in  a  relaxation  scheme  for  classifying  the  line 
segments. 


6 .  Extensions  and  conclusions 

The  following  extensions  that  may  improve  the  results  of 
the  linking  process  are  suggested. 

1)  Include  the  distribution  (e.g.  standard  deviation)  of 
the  object  side  gray  level  along  the  linking  line  in  the  figure 
of  merit.  There  are  rare  cases  in  which  the  gray  level  along 
the  ok  ject  side  of  the  linking  line  changes  in  such  a  way  as 

t<,  produce  an  average  gray  level  similar  to  that  of  the  pair, 
resuJting  from  gray  levels  that  are  partly  too  high  and  partly 
too  low  and  hence  not  actually  similar. 

2)  The  same  as  (1)  but  for  the  pair  of  lines  themselves. 

3)  The  linking  line  should  not  intersect  other  lines. 

This  paper  has  presented  an  algorithm  for  linking  compatible 
linear  features.  The  linking  process  is  based  on  the  degree  of 
similarity  of  the  gray  levels  of  the  object  sides  of  the  pairs 
and  the  linking  lines,  the  angles  between  them,  geometrical 
conditions,  and  also  the  estimated  object  gray  level.  The  algo¬ 
rithm  was  evaluated  using  lines  extracted  from  real  images  using 
an  edge  detection  and  enhancement  process.  The  results  were  seen 
to  be  generally  reasonable  and  the  result  of  applying  the  process 
was  a  new  image  representation  that  was  more  useful  for  further 
processing  than  the  initial  edge  image.  This  process  should  be 
especially  useful  for  defining  the  degrees  of  compatibility  of 
the  line  segments  in  a  relaxation  scheme  for  classifying  the 
feature  segments. 
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Examples  of  different  orientations  of  geometrically 
compatible  lines. 


Linear  function  for  calculation  of  the  second 
geometrical  measure. 
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Figure  12.  The  histogram  of  the  first  figure  of  merit 


S  =10,  suburban  scene, 
max 


Figure  13.  Line  linking:  FMl=2.3,  smax~l°* 


Figure  14.  Line  linking:  FM1=2.0,  smax=^0* 
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Figure  17.  The  histogram  of  the  first  figure  of  merit:  S 

suburban  scene.  "  ma* 


12, 


Figure  19.  Line  linking:  FMl^2.0,  S 


12. 


Figure  21.  Line  linking:  FMl=0.1,  Smax=12. 
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Figure  22.  The  histogram  of  the  first  figure  of  merit: 


SmaX=15,  suburban  scene. 


Figure  27.  A  portion  of  Lorton  Reformatory. 


Figure  28.  The  extracted  lines. 
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Figure  2  -  The  histogram  of  the  first  figure  of  merit 
^^=10,  Lorton  scene. 
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Figure  34.  Histogram  of  the  second  figure  of  merit: 
Smax=10,  suburban  scene. 
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Figure  39.  Histogram  of  the  second  figure  of  merit: 


Smax=10,  Lorton  scene. 
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Figure  44.  The  histogram  of  the  second  figure  of  merit: 


Smax=12,  suburban  scene. 
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Figure  49.  The  histogram  of  the  second  figure  of  merit; 


Sm=  =15,  suburban  scene. 
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